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Toward  a  Descriptive  Model  of 
Post-Implementation   Evaluation 


Abstract 


Strategies  for  evaluating  computer-based  information  systems  (CBISs)  recommended  in  the 
information  systems  (IS)  literature  are  generally  based  upon  formal,  quantitative  models  of 
evaluation.  However,  evidence  suggests  that  IS  professionals  frequently  omit  formal,  quantitative 
evaluation  of  CBISs  and  rely  instead  on  informal,  qualitative  evaluation.  If  formal,  quantitative 
models  of  CBIS  evaluation  are  of  value,  why  are  they  infrequently  used  by  their  intended 
beneficiaries? 

Distinguishing  between  uncertainty  and  equivocality  provides  insight  into  why  IS 
professionals  might  omit  formal,  quantitative  evaluation.  Uncertainty  is  the  absence  of  information, 
while  equivocality  is  information  that  is  unclear,  conflicting  or  paradoxical.  Evaluation  designed  to 
reduce  uncertainty  uses  formal  processes  and  methods,  defined  organizational  roles  and 
responsibilities,  quantifiable  criterion,  and  objective  data.  Evaluation  designed  to  reduce 
equivocality  uses  informal  processes  and  methods,  negotiated  roles  and  responsibilities,  qualitative 
criterion,  and  subjective  data.  One  explanation  why  IS  professionals  frequently  omit  formal, 
quantitative  evaluation  of  CBIS  may  be  that  such  procedures  are  not  helpful  in  reducing 
equivocality. 
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Toward  a  Descriptive  Model  of 
Post-Implementation  Evaluation 

An  organization  implements  a  computer-based  information  system  (CBIS).  Later,  someone 
asks  if  the  CBIS  is  a  success  or  failure.  Does  it  contribute  to  organizational  goals?  Should  it  be 
maintained,  expanded,  replaced,  or  abandoned? 

Evidence  suggests  that  the  inability  to  measure  and  evaluate  productivity  gains  is  a  major 
obstacle  to  investment  in  CBISs  (Blacker  and  Brown,  1988;  Strassman,  1985).  Controversy  over 
measuring  productivity  contributions  from  new  technology  has  resulted  in  increasing  skepticism 
regarding  the  benefits  of  CBISs  (Bowen,  1986;  Business  Week,  1988).  One  approach  to 
understanding  the  controversy  over  productivity  measurement  is  to  reexamine  the  methods  and 
assumptions  of  existing  CBIS  evaluation  models. 

Researchers  have  long  recognized  the  importance  and  complexity  of  evaluating  CBISs.  As 
a  result,  a  number  of  formal,  quantitative  methods  for  evaluating  CBISs  have  been  suggested 
(e.g.,  King  and  Schrems,  1978;  King  and  Epstein,  1983;  Piepta  and  Anderson,  1987; 
Schwuchow,  1977).  However,  evidence  suggests  that  formal,  quantitative  methods  (e.g.,  cost- 
benefit  analysis)  for  evaluating  CBISs  are  relatively  infrequently  used  (Greiner,  Leitch,  and 
Barnes,  1979;  Hogue  and  Watson,  1984),  and  are  considered  of  dubious  value  by  many 
information  systems  (IS)  researchers  and  practitioners  (Keen,  1981;  Hirschheim  and  Smithson, 
1988;  Zmud  and  Apple,  1989).  The  schism  between  the  recommendations  for  evaluation  found  in 
the  IS  literature  and  descriptions  of  evaluation  practice  suggests  an  obvious  question.  If  formal, 
quantitative  methods  for  CBIS  evaluation  are  of  value,  why  are  they  infrequently  used  by  their 
intended  beneficiaries?  This  purpose  of  this  paper  is  to  develop  a  model  that  provides  insight  into 
this  and  related  questions. 

CBIS  evaluation  is  herein  defined  as  the  process  of  determining  how  a  CBIS  impacts  and  is 
impacted  by  an  organization.  Several  assumptions  are  implicit  in  this  definition.  First,  that  a  CBIS 
has  been  implemented,  meaning  that  evaluation  is  a  post-implementation  activity.  Evaluation  is 
therefore  identified  as  distinct  from  feasibility  analysis  (e.g.,  Caddell,  1985)  and  a  priori 
justification  of  CBISs  (e.g.,  Bozcany,  1983).  Second,  it  is  assumed  that  organizations  both  create 
and  are  created  by  CBISs  (Markus,  1984).  Evidence  suggests  that  implementing  a  CBIS  can 
trigger  complex,  often  unanticipated  chains  of  events  in  organizations  (Barley;  1986;  Markus  and 
Robey,  1988).  These  chains  of  events  ultimately  mean  that  organizations  shape  and  are  shaped  by 
CBISs.  Finally,  it  is  assumed  that  evaluation  can  be  either  formal  (e.g.  cost-benefit  analysis)  or 
informal  (e.g.  a  conversation  between  two  IS  managers  over  lunch).  Relaxing  the  typical  definition 
of  evaluation  as  a  formal,  quantitative  process  permits  building  a  more  descriptive  framework  that 
recognizes  both  planned  and  unplanned,  and  formal  and  informal  evaluation. 


Post-Implementation  Evaluation 

4 

This  paper  develops  a  model  that  explains  why  IS  professionals  frequently  omit  formal, 
quantitative  CBIS  evaluation,  relying  instead  on  less  formal,  qualitative  methods.  The  presentation 
of  this  model  is  organized  as  follows.  First,  uncertainty  is  distinguished  from  equivocality,  and 
expected  differences  in  uncertainty-reducing  and  equivocality-reducing  CBIS  evaluation  are 
identified.  Second,  a  descriptive,  contingency  model  of  CBIS  evaluation  is  proposed  that  relates 
the  usefulness  of  uncertainty  and  equivocality-reducing  evaluation  to  relationships  between 
organizational  actors.  The  paper  concludes  with  a  discussion  of  the  implications  of  the  model  for 
CBIS  evaluation  research. 

Uncertainty  and  Equivocality-Reducing  CBIS  Evaluation 

Uncertainty  and  Equivocality 

A  useful  dichotomy  in  considering  CBIS  evaluation  is  the  distinction  between  uncertainty 
and  equivocality.  Uncertainty  is  the  absence  of  information  (Miller  and  Frick,  1949;  Daft  and 
Lengel,  1986).  As  information  increases,  uncertainty  decreases.  The  game  of  20  questions 
illustrates  uncertainty  and  uncertainty  reduction.  A  questioner  receives  yes-no  answers  to  questions 
intended  to  identify  an  unknown  object  as  either  animal,  vegetable,  or  mineral  (Taylor  and  Faust, 
1952;  Bendig,  1953;  Daft  and  Lengel  1986).  Uncertainty  is  eliminated  when  the  object  is  correctly 
identified.  In  management  tasks  characterized  by  uncertainty,  managers  are  able  to  ask  questions, 
and  get  answers  that  permit  problem  solving.  Organizational  processes  can  be  structured  to  reduce 
uncertainty  through  the  use  of  rules  and  regulations  and  through  the  creation  of  formal,  structured 
IS  (Daft  and  Lengel,  1986). 

In  contrast,  equivocality  involves  interpreting  data  that  is  unclear,  conflicting,  or 
paradoxical  (Daft  and  Macintosh,  1981;  Weick,  1979).  The  sentence,  "I  saw  the  man  on  the  hill 
with  the  telescope,"  (Simon,  1982,  p.  93)  is  equivocal:  multiple  interpretations  are  possible.  Do  I 
have  the  telescope,  or  does  the  man  on  the  hill?  Is  the  telescope  merely  on  the  hill  and  not  in  the 
man's  hand?  Managers  deal  with  'men  on  hills  with  telescopes'  regularly,  and  must  make  sense  of 
such  equivocality.  Daft,  Lengel,  and  Trevino  (1987,  p.  357)  observe  that  in  equivocal 
environments,  "Managers  are  not  certain  what  questions  to  ask,  and  if  questions  are  posed  there  is 
no  store  of  objective  data  to  provide  an  answer." 

Fundamental  to  the  process  of  managing  equivocality  is  "sense-making,"  which  involves 
exchanges  between  managers  intended  to  reduce  equivocality  and  create  a  shared  interpretation  that 
can  direct  future  events  (Weick,  1979).  When  facing  equivocality,  managers  build  a  shared 
interpretation,  and  "enact"  a  solution,  rather  than  relying  on  data  gathering  activities  to  direct  events 
(Daft  and  Weick,  1984).  The  processes  of  sense-making  and  enactment  involve  exchanging 
subjective  opinions,  managing  multiple  perspectives,  and  proactively  shaping  environments 
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(Smircich  and  Stubbart,  1985).  Equivocality  is  reduced  by  managing  and  generating  both  events 
and  interpretations  of  events. 

CBIS  Evaluation  as  Uncertainty-reducing  Activity 

Approaches  to  CBIS  evaluation  contain  underlying  assumptions  as  to  whether  evaluation 
processes  should  be  designed  primarily  to  reduce  uncertainty  or  equivocality.  Existing  research  on 
assessing  CBIS  impact  generally  views  evaluation  as  an  uncertainty-reducing  process  based  upon 
formal,  objective  data  collection  and  information  processing  (e.g.  Hamilton  and  Chervany,  1981a, 
1981b).  Viewing  CBIS  evaluation  as  a  data  collection  activity  leads  to  an  evaluation  process 
focused  on  gathering  and  processing  data  to  reduce  and  eliminate  uncertainty  about  CBIS  impact. 
CBIS  evaluation  methods  based  upon  formal,  objective  data  collection  include  cost-benefit  analysis 
(e.g.,  Emery,  1982;  King  and  Schrems,  1978),  user  surveys  (e.g.,  Miller  and  Doyle,  1987; 
Rushinek  and  Rushinkek,  1983),  measures  of  computer  usage  (e.g.,  Ferrari  1978;  Hiltz  and 
Turoff,  1981),  and  the  use  of  "objective"  data  sources  external  to  the  implementing  organization 
(e.g.,  Banker  and  Kauffman,  1988). 

Uncertainty-reducing  CBIS  should  be  recognizable  by  the  existence  of  organizational 
processes  oriented  towards  formal  data  gathering  and  information  processing.  Figure  1  describes 
organizational  processes  that  should  exist  when  CBIS  evaluation  is  perceived  as  an  uncertainty- 
reducing  activity.  In  general,  uncertainty-reducing  CBIS  evaluation  should  rely  on  formal 
evaluation  processes  and  procedures,  should  utilize  the  defined  authority  structure  of  the 
organization,  should  rely  on  quantifiable  measures  of  the  system,  and  should  be  based  upon 
objective,  verifiable  data.  In  addition,  CBIS  evaluation  designed  to  reduce  uncertainty  is  likely  to 
focus  on  measuring  expected,  anticipated  effects,  rather  than  exploring  unplanned,  unexpected 
effects  of  CBISs. 


Insert  Figure  1  about  here 


CBIS  Evaluation  as  Equivocality-reducing  Activity 

An  alternative  evaluative  perspective  is  to  view  assessing  IS  impact  as  a  sense-making 
process  (Weick,  1985).  Viewing  CBIS  evaluation  as  sense-making  suggests  that  determining  the 
impact  of  an  information  system  requires  interpreting  conflicting,  ambiguous  information,  and  may 
involve  building  and  enacting  shared  interpretations  of  events  to  resolve  equivocality. 
Consequently,  CBIS  evaluation  as  sense-making  leads  to  an  evaluation  process  largely  focused  on 
exchanging  subjective  opinions  and  beliefs,  rather  than  gathering  formal,  objective  data. 
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Figure  2  describes  organizational  processes  that  should  exist  when  CBIS  evaluation  is 
conducted  as  an  equivocality-reducing  activity.  In  general,  equivocality-reducing  CBIS  evaluation 
should  rely  on  informal  (rather  than  planned)  meetings  and  discussions,  on  negotiated  (rather  than 
assigned)  roles  and  responsibilities,  on  qualitative  (rather  than  quantitative)  dimensions  of  system 
success,  and  on  subjective  (rather  than  objective)  data.  In  addition,  equivocality-reducing  CBIS 
evaluation  is  more  likely  to  consider  unplanned  impacts  (e.g.  changes  in  social  relationships). 


Insert  Figure  2  about  here 
A  Model  of  CBIS  Evaluation 


When  are  organizations  likely  to  undertake  uncertainty-reducing  versus  equivocality- 
reducing  CBIS  evaluation?  Two  characteristics  that  may  be  useful  in  predicting  the  evaluative 
approach  used  are:  (1)  the  extent  of  agreement  among  organizational  actors  as  to  CBIS-related 
goals,  and  (2)  the  extent  of  agreement  as  to  whether  an  implemented  CBIS  achieves  system-related 
goals  (i.e.,  whether  the  CBIS  provides  the  means  for  achieving  goals).  The  extent  of  agreement 
on  means  and  goals  is  likely  to  influence  the  importance  of  uncertainty-reducing  versus 
equivocality-reducing  activites,  and  to  thereby  influence  CBIS  evaluation. 

Organizational  actors  may  have  differing,  conflicting  goals  with  respect  to  an  implemented 
CBIS  (Kling,  1987;  Kling,  1980).  Actors  may  value  CBISs  as  a  means  of  achieving  functional 
objectives  (e.g.,  reducing  costs),  as  a  symbol  of  the  importance  of  an  individual  or  group  within 
the  organization,  or  as  a  signal  of  commitment  to  particular  organizational  ideologies  (Feldman  and 
March,  1981;  Robey  and  Markus,  1984).  Agreement  among  actors  on  goals  reduces  equivocality, 
since  objectives  can  be  assumed,  and  need  not  be  constructed  through  sense-making  and  enactment 
processes. 

When  organizational  actors  disagree  as  to  CBIS-related  goals,  equivocality  will  be  high.  As 
organizational  actors  move  towards  disagreement  on  goals,  CBIS  evaluation  will  likely  move 
towards  equivocality-reducing  processes.  Consequently,  evaluation  processes  are  likely  to  assume 
the  characteristics  of  equivocality-reducing  evaluation  stated  in  Figure  2. 

Actors  may  also  disagree  as  to  whether  an  implemented  CBIS  achieves  desired  goals  (i.e., 
does  the  CBIS  provide  the  means  for  achieving  goals?).  For  example,  actors  may  agree  that 
reducing  costs  is  desirable,  but  may  disagree  as  to  whether  an  implemented  CBIS  has  achieved 
cost  savings.  Agreement  among  actors  on  means  for  achieving  goals  (e.g.,  the  system  does  reduce 
costs)  decreases  uncertainty. 

When  organizational  actors  disagree  on  means  for  achieving  CBIS-related  goals, 
uncertainty  will  be  high.  As  organizational  actors  move  towards  disagreement  on  means,  CBIS 
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evaluation  will  likely  move  towards  objective  data  gathering  and  information  processing  designed 
to  reduce  uncertainty  about  CBIS  impact.  Consequently,  as  organizational  actors  move  toward 
disagreement  on  means,  evaluation  processes  are  likely  to  assume  the  characteristics  of  uncertainty- 
reducing  evaluation  stated  in  Figure  1. 

Figure  3  is  a  descriptive  model  of  CBIS  evaluation  that  summarizes  the  hypothesized 
relationships  between  agreement  on  goals  and  means,  and  CBIS  evaluation  processes.  When 
agreement  on  both  goals  and  means  for  achieving  goals  is  high  (cell  1),  evaluation  is  trivial,  since 
actors  agree  both  as  to  goals,  and  as  to  whether  the  CBIS  achieves  agreed-upon  goals.  When 
agreement  on  CBIS-related  means  is  low,  but  agreement  on  goals  is  high  (cell  2),  uncertainty  will 
be  high,  and  evaluation  will  be  constructed  primarily  to  reduce  uncertainty,  resulting  in  formal, 
quantitative  evaluation.  When  agreement  on  CBIS-related  goals  is  low,  but  agreement  on  means  is 
high  (cell  3),  equivocality  will  be  high,  and  evaluation  will  be  undertaken  to  reduce  equivocality, 
resulting  in  informal,  qualitative  evaluation.  When  agreement  on  both  goals  and  means  is  low  (cell 
4),  uncertainty  and  equivocality  are  high,  and  CBIS  evaluation  is  likely  to  employ  both  uncertainty- 
reducing  and  equivocality-reducing  evaluation  processes.  In  such  cases,  evaluation  is  likely  to 
employ  both  formal  and  informal  processes,  defined  and  negotiated  roles  and  responsibilities,  and 
qualitative  and  quantitative  criteria. 


Insert  Figure  3  about  here 

Discussion  and  Conclusion 

Most  existing  research  views  CBIS  evaluation  as  an  uncertainty-reducing  activity. 
However,  Such  a  perspective  does  not  explain  the  infrequent  use  of  formal  data  collection  activities 
evidenced  in  surveys  of  evaluation  practice.  One  explanation  why  IS  professionals  often  omit 
formal  CBIS  evaluation  and  rely  instead  on  informal,  subjective  interpretations  of  system  impact 
may  be  that  formal  CBIS  evaluation  is  of  little  value  in  reducing  equivocality. 

Ultimately,  the  goal  of  investigating  CBIS  evaluation  is  to  offer  prescriptions  for  improving 
productivity  measurement.  However,  existing  methods  for  evaluating  CBISs  largely  ignore 
evaluation  as  an  equivocality-reducing  process.  The  intent  of  this  paper  is  to  legitimize  informal, 
unplanned,  equivocality-reducing  evaluation,  by  recognizing  that  informal  evaluation  may  be  of 
greater  value  than  formal  evaluation  under  certain  circumstances.  Ideally,  legitimizing  equivocality- 
reducing  evaluation  will  lead  to  normative  and  prescriptive  evaluation  approaches  that  take 
seriously  the  subjective,  informal,  impressionistic  evaluations  considered  largely  irrelevant  and 
uninformative  by  previous  CBIS  evaluation  research. 
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Figure   1 
Characteristics  of  Uncertainty-reducing  CBIS  Evaluation 

*  Emphasis  on  formal  processes  and  formal  evaluation,  e.g.  formal  reports,  formal 
meetings. 

*  Use  of  formal,  defined  organizational  authority  structure  to  manage  evaluation,  e.g. 
official  titles  and  roles. 

*  Emphasis  on  quantifiable  and  direcdy  measurable  aspects  of  system,  e.g.  costs,  system 
usage,  etc.. 

*  Use  of  objective,  verifiable  data  sources  and  methods,  e.g.  user  surveys,  cost-benefit 
analysis,  system  logs,  etc.. 

*  Emphasis  on  planned  CBIS  impact,  e.g.  cost  savings,  administrative  convenience. 


Figure  2 
Characteristics  of  Equivocality-reducing  CBIS  Evaluation 


*  Emphasis  on  unplanned  processes  and  informal  evaluation,  e.g.  unplanned  meetings, 
informal  discussion. 

*  Use  of  informal  organizational  authority  structure  to  manage  evaluation,  e.g.  negotiation 
of  roles  and  responsibilities. 

*  Use  of  subjective  opinions,  impressionistic  "evidence"  and  experiential,  interpretive 
"methods,"  e.g.  in-depth  interviews,  systematic  reflection. 

*  Consideration  of  unplanned  CBIS  impact,  e.g.  social  relationships,  unexpected 
consequences. 


Figure  3 
A  Descriptive  Model  of  CBIS  Evaluation 
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